Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks

The United States COVID-19 Forecast Hub dataset

Academic researchers, government agencies, industry groups, and individuals have produced forecasts at an unprecedented scale during the COVID-19 pandemic. To leverage these forecasts, the United States Centers for Disease Control and Prevention (CDC) partnered with an academic research lab at the University of Massachusetts Amherst to create the US COVID-19 Forecast Hub. Launched in April 2020, the Forecast Hub is a dataset with point and probabilistic forecasts of incident cases, incident hospitalizations, incident deaths, and cumulative deaths due to COVID-19 at county, state, and national, levels in the United States. Included forecasts represent a variety of modeling approaches, data sources, and assumptions regarding the spread of COVID-19. The goal of this dataset is to establish a standardized and comparable set of short-term forecasts from modeling teams. These data can be used to develop ensemble models, communicate forecasts to the public, create visualizations, compare models, and inform policies regarding COVID-19 mitigation. These open-source data are available via download from GitHub, through an online API, and through R packages

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks

Prospective observational cohort study on grading the severity of postoperative complications in global surgery research

Background The Clavien–Dindo classification is perhaps the most widely used approach for reporting postoperative complications in clinical trials. This system classifies complication severity by the treatment provided. However, it is unclear whether the Clavien–Dindo system can be used internationally in studies across differing healthcare systems in high- (HICs) and low- and middle-income countries (LMICs). Methods This was a secondary analysis of the International Surgical Outcomes Study (ISOS), a prospective observational cohort study of elective surgery in adults. Data collection occurred over a 7-day period. Severity of complications was graded using Clavien–Dindo and the simpler ISOS grading (mild, moderate or severe, based on guided investigator judgement). Severity grading was compared using the intraclass correlation coefficient (ICC). Data are presented as frequencies and ICC values (with 95 per cent c.i.). The analysis was stratified by income status of the country, comparing HICs with LMICs. Results A total of 44 814 patients were recruited from 474 hospitals in 27 countries (19 HICs and 8 LMICs). Some 7508 patients (16·8 per cent) experienced at least one postoperative complication, equivalent to 11 664 complications in total. Using the ISOS classification, 5504 of 11 664 complications (47·2 per cent) were graded as mild, 4244 (36·4 per cent) as moderate and 1916 (16·4 per cent) as severe. Using Clavien–Dindo, 6781 of 11 664 complications (58·1 per cent) were graded as I or II, 1740 (14·9 per cent) as III, 2408 (20·6 per cent) as IV and 735 (6·3 per cent) as V. Agreement between classification systems was poor overall (ICC 0·41, 95 per cent c.i. 0·20 to 0·55), and in LMICs (ICC 0·23, 0·05 to 0·38) and HICs (ICC 0·46, 0·25 to 0·59). Conclusion Caution is recommended when using a treatment approach to grade complications in global surgery studies, as this may introduce bias unintentionally

Critical care admission following elective surgery was not associated with survival benefit: prospective analysis of data from 27 countries

This was an investigator initiated study funded by Nestle Health Sciences through an unrestricted research grant, and by a National Institute for Health Research (UK) Professorship held by RP. The study was sponsored by Queen Mary University of London

The surgical safety checklist and patient outcomes after surgery: a prospective observational cohort study, systematic review and meta-analysis

© 2017 British Journal of Anaesthesia Background: The surgical safety checklist is widely used to improve the quality of perioperative care. However, clinicians continue to debate the clinical effectiveness of this tool. Methods: Prospective analysis of data from the International Surgical Outcomes Study (ISOS), an international observational study of elective in-patient surgery, accompanied by a systematic review and meta-analysis of published literature. The exposure was surgical safety checklist use. The primary outcome was in-hospital mortality and the secondary outcome was postoperative complications. In the ISOS cohort, a multivariable multi-level generalized linear model was used to test associations. To further contextualise these findings, we included the results from the ISOS cohort in a meta-analysis. Results are reported as odds ratios (OR) with 95% confidence intervals. Results: We included 44 814 patients from 497 hospitals in 27 countries in the ISOS analysis. There were 40 245 (89.8%) patients exposed to the checklist, whilst 7508 (16.8%) sustained ≥1 postoperative complications and 207 (0.5%) died before hospital discharge. Checklist exposure was associated with reduced mortality [odds ratio (OR) 0.49 (0.32–0.77); P\u3c0.01], but no difference in complication rates [OR 1.02 (0.88–1.19); P=0.75]. In a systematic review, we screened 3732 records and identified 11 eligible studies of 453 292 patients including the ISOS cohort. Checklist exposure was associated with both reduced postoperative mortality [OR 0.75 (0.62–0.92); P\u3c0.01; I2=87%] and reduced complication rates [OR 0.73 (0.61–0.88); P\u3c0.01; I2=89%). Conclusions: Patients exposed to a surgical safety checklist experience better postoperative outcomes, but this could simply reflect wider quality of care in hospitals where checklist use is routine